Papers with contextual word embeddings
Multilingual, Multi-scale and Multi-layer Visualization of Intermediate Representations (D19-3)
Copied to clipboard
| Challenge: | Currently, the main alternatives to deal with sequences are Recurrent Neural Networks (RNN) architectures and the Transformer. |
| Approach: | They propose a web-based tool that visualizes the sentence and token representations of RNNs and Transformer architectures at the sentence level. |
| Outcome: | The proposed visualization tool analyses gender inequalities in contextual word embeddings and the common language representation in a multilingual machine translation system. |
BERTrade: Using Contextual Embeddings to Parse Old French (2022.lrec-1)
Copied to clipboard
| Challenge: | a growing interest in digital humanities for automatic processing and annotation of historical texts is generating new models for historical languages. |
| Approach: | They use POS-tagging and dependency parsing to evaluate contextual word embedding models . Old French is one of the historical languages for which they have the largest amount of syntactically annotated data . |
| Outcome: | The proposed model can be used to improve performance in Old French, the authors show . they use POS-tagging and dependency parsing to evaluate the model's quality . |
Exploiting Position and Contextual Word Embeddings for Keyphrase Extraction from Scientific Papers (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing methods for keyphrase extraction are either supervised or unsupervised. |
| Approach: | They propose an unsupervised algorithm that exploits contextual word embeddings and positional information to create a biased PageRank. |
| Outcome: | The proposed algorithm outperforms previous approaches and strong baselines on five benchmark datasets. |
Abusive language in Spanish children and young teenager’s conversations: data preparation and short text classification with contextual word embeddings (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing studies on how to automatically detect abusive short texts are gaining interest in the natural language processing community. |
| Approach: | They propose to use a contextual word embedding model to automatically detect abusive short texts for Spanish language. |
| Outcome: | The proposed model outperforms classical methods in the detection of abusive short texts for the spanish language. |
Semantics or spelling? Probing contextual word embeddings with orthographic noise (2024.findings-acl)
Copied to clipboard
| Challenge: | Pretrained language models (PLMs) are used to generate contextual word embeddings . linguistics research has focused on semantic information in hidden states . |
| Approach: | They investigate whether a single character swap in the input word will not affect the resulting representation . they find that PLM-derived contextual word embeddings are highly sensitive to noise . |
| Outcome: | The results show that the PLM-derived representations are highly sensitive to noise . the fewer tokens used to represent a word at input, the more sensitive their CWE is . |
Putting Evaluation in Context: Contextual Embeddings Improve Machine Translation Evaluation (P19-1)
Copied to clipboard
| Challenge: | Existing evaluation metrics are limited and can be easily portable to new languages. |
| Approach: | They propose a simple unsupervised metric and additional supervised metrics which rely on contextual word embeddings to encode the translation and reference sentences. |
| Outcome: | The proposed model outperforms existing metrics on the WMT 2017 dataset and is more accurate than existing models. |
A Challenge Set and Methods for Noun-Verb Ambiguity (D18-1)
Copied to clipboard
| Challenge: | English part-of-speech taggers make egregious errors related to noun-verb ambiguity, despite having achieved 97%+ accuracy on the WSJ Penn Treebank since 2002. |
| Approach: | They propose to use a WSJ dataset to identify 30,000 examples of noun-verb ambiguity . they find that english part-of-speech taggers make egregious errors related to nouns and verbs . |
| Outcome: | The proposed model improves on the WSJ Penn Treebank by 14% and 52% relative to the previous model. |
Obtaining Better Static Word Embeddings Using Contextual Embedding Models (2021.acl-long)
Copied to clipboard
| Challenge: | Recent contextual word embeddings have prohibitively high computational cost in many use-cases and are hard to interpret. |
| Approach: | They propose a distillation method which is an extension of CBOW-based training and improves computational efficiency of NLP applications. |
| Outcome: | The proposed method outperforms existing models and existing models in terms of quality and performance. |
Spying on Your Neighbors: Fine-grained Probing of Contextual Embeddings for Information about Surrounding Words (2020.acl-main)
Copied to clipboard
| Challenge: | a suite of probing tasks test contextual embeddings for encoding of information about surrounding words . authors: little is known about what information embeddables encode about the context words encode . a recent study shows that contextual embeds can be powerful for many tasks . |
| Approach: | They propose probing tasks that enable fine-grained testing of contextual embeddings . they examine popular contextual encoders and find that each encodes contextual information across tokens a little different . |
| Outcome: | The proposed probing tasks show that word embeddings encode information about words . the tests show that the encoded information is encoded across tokens with near-perfect recoverability . |
Disentangling language change: sparse autoencoders quantify the semantic evolution of indigeneity in French (2025.naacl-long)
Copied to clipboard
Jacob A. Matthews, Laurent Dubreuil, Imane Terhmina, Yunci Sun, Matthew Wilkens, Marten Van Schijndel
| Challenge: | Existing methods to measure semantic change with contextual word embeddings (CWEs) are not suitable for highly imbalanced datasets and pose challenges for interpretation. |
| Approach: | They propose an interpretable, feature-level approach to analyzing language change using k-sparse autoencoders to trace the semantic evolution of the term "indigène(s)" between 1825 and 1950. |
| Outcome: | The proposed approach can learn interpretable features from over 210,000 CWEs generated using sentences from the French National Library. |
Self-Attention Architectures for Answer-Agnostic Neural Question Generation (P19-1)
Copied to clipboard
| Challenge: | Neural architectures based on self-attention have attracted interest from the research community . a recent study examined the performance of Transformers on a task of Neural Question Generation . |
| Approach: | They propose to adapt Transformers to a task of Neural Question Generation without constraining the model to focus on a specific answer passage. |
| Outcome: | The proposed architectures have obtained significant improvements over the state-of-the-art in several tasks. |
Uniform Information Density and Syntactic Reduction: Revisiting *that*-Mentioning in English Complement Clauses (2025.emnlp-main)
Copied to clipboard
| Challenge: | Uniform Information Density (UID) hypothesis suggests that speakers exploit this variability to maintain a consistent rate of information transmission during language production. |
| Approach: | They propose that speakers exploit this variability to maintain a consistent rate of information transmission during language production. |
| Outcome: | The proposed hypothesis replicates the established relationship between information density and *that*-mentioning . |